Using a national surgical database to predict complications following posterior lumbar surgery and comparing the area under the curve and F1-score for the assessment of prognostic capability.

BACKGROUND: With spinal surgery rates increasing in North America, models that are able to accurately predict which patients are at greater risk of developing complications are highly warranted. However, the previously published methods which have used large, multi-centre databases to develop their prediction models have relied on the receiver operator characteristics curve with the associated area under the curve (AUC) to assess their model's performance. Recently, it has been found that a precision-recall curve with the associated F1-score could provide a more realistic analysis for these models. PURPOSE: To develop a logistic regression (LR) model for the prediction of complications following posterior lumbar spine surgery and to then assess for any difference in performance of the model when using the AUC versus the F1-score. STUDY DESIGN: Retrospective review of a prospective cohort. PATIENT SAMPLE: The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) registry was used. All patients that underwent posterior lumbar spine surgery between 2005 to 2016 with appropriate data were included. OUTCOME MEASURES: Both the AUC and F1-score were utilized to assess the prognostic performance of the prediction model. METHODS: In order to develop the LR model used to predict a complication during or following spine surgery, 19 variables were selected by three orthopedic spine surgeons from the NSQIP registry. Two datasets were developed for this analysis: (1) an imbalanced dataset, which was taken directly from the NSQIP registry, and (2) a down-sampled set. The purpose of the down-sampled set was to balance the data in order to evaluate whether balancing the data had an effect on model performance. The AUC and F1-score were applied to both of these datasets. RESULTS: Within the NSQIP database, 52,787 spine surgery cases were identified of which only 10% of these cases had complications during surgery. Applying the LR model showed a large difference between the AUC (0.69) and the F1 score (0.075) on the imbalanced dataset. However, no major differences existed between the AUC and F1-score when the data was balanced and the LR model was reapplied (0.69 and 0.62, AUC and F1-score, respectively). CONCLUSIONS: The F1-score detected a drastically lower performance for the prediction of complications when using the imbalanced data, but detected a performance similar to the AUC level when balancing techniques were utilized for the dataset. This difference is due to a low precision score when many false positive classifications are present, which is not identified when using the AUC value. This lowers the utility of the AUC score, as many of the datasets used in medicine are imbalanced. Therefore, we recommend using the F1-score on large, prospective databases when the data is imbalanced with a large amount of true negative classifications.

Development of an unsupervised machine learning algorithm for the prognostication of walking ability in spinal cord injury patients.

BACKGROUND CONTEXT: Traumatic spinal cord injury can have a dramatic effect on a patient's life. The degree of neurologic recovery greatly influences a patient's treatment and expected quality of life. This has resulted in the development of machine learning algorithms (MLA) that use acute demographic and neurologic information to prognosticate recovery. The van Middendorp et al. (2011) (vM) logistic regression (LR) model has been established as a reference model for the prediction of walking recovery following spinal cord injury as it has been validated within many different countries. However, an examination of the way in which these prediction models are evaluated is warranted. The area under the receiver operators curve (AUROC) has been consistently used when evaluating model performance, but it has been shown that AUROC overemphasizes the most common event resulting in an inaccurate assessment when the data are imbalanced. Furthermore, there is evidence that the use of more advanced MLA, such as an unsupervised k-means model, may show superior performance compared to LR as they can handle a larger number of features. PURPOSE: The first objective of the study was to assess the performance of both an unsupervised MLA and LR model with complete admission neurologic information against the vM and Hicks models. Second, a comparison between the accuracy of the AUROC and the F1-score will be made to determine which method is superior for the assessment of diagnostic performance of prediction models on large-scale datasets. STUDY DESIGN: Retrospective review of a prospective cohort study. PATIENT SAMPLE: The Rick Hansen Spinal Cord Injury Registry (RHSCIR) was used in this study. All patients enrolled between 2004 and 2017 with complete neurologic examination and Functional Independence Measure outcome data at ≥1 year follow-up or who could walk at discharge were included. The prognostic variables included age (dichotomized at ≥65 years old); American Spinal Injury Association Impairment Scale (AIS) grade; and individual motor, light touch, and pinprick score from L2 to S1. OUTCOME MEASURES: The Functional Independence Measure locomotor score was used to assess independent walking ability at discharge or 1-year follow-up. METHODS: An unsupervised MLA with k=2 was chosen in order to identify a "walk" cluster and a "not walk" cluster. Model performance was assessed through the development of a receiver operating characteristic curve with associated AUROC and a precision-recall curve with associated F1-score. The study and the RHSCIR are supported by funding from Health Canada, Western Economic Diversification Canada, and the Governments of Alberta, British Columbia, Manitoba, and Ontario. These funders had no role in the study or study reporting and the authors have no conflicts of interest to report. RESULTS: No clinically relevant differences were found between with the use of an unsupervised MLA with a greater amount of initial neurologic information compared to the established standards for any AIS classification. Although demonstrated for all separate AIS classifications, most notably, the AUROC for the vM (0.78) and Hicks models (0.76) were found to be superior to that of the new LR model (0.72); however, the vM and Hicks models had more than double the amount of false negative classifications compared to the LR. The F1-scores between these three models were also found to be different but with the vM and Hicks models being lower than the LR (0.85, 0.81, and 0.89, respectively). CONCLUSIONS: No clinically relevant differences were found between the use of an unsupervised MLA with complete admission neurologic information compared to the previously validated standards; however, when comparing the performance of the AUROC and F1-score, the AUROC showed inaccurate prognostic performance when there was an imbalance toward a greater amount of false negatives. Importantly, the F1-score did not succumb to this imbalance. As AUROC has been used as the standard when evaluating performance of prediction models, consideration as to whether this is the most appropriate method is warranted. Future work should focus on comparing AUROC and F1-scores with other previously validated models.

Carbohydrate Replacement of Rice or Potato with Lentils Reduces the Postprandial Glycemic Response in Healthy Adults in an Acute, Randomized, Crossover Trial.

Background: The postprandial blood glucose response (PBGR) following carbohydrate replacement of high-glycemic index (GI) foods with pulses, in a mixed meal, has not been accurately defined. Objective: We aimed to determine the extent to which PBGR and relative glycemic response (RGR) are lowered when half of the available carbohydrate (AC) from rice or potato is replaced with cooked lentils. Methods: Using a crossover design, 2 groups of 24 healthy adults randomly consumed 50 g AC from control white rice alone [mean ± SD body mass index (BMI, in kg/m2): 24.3 ± 0.5; mean ± SD age: 27.7 ± 1.2 y], instant potato alone (BMI: 24.0 ± 0.5; age: 27.4 ± 1.2 y), or the same starch source in a 50:50 AC combination with each of 3 types of commercially available lentils (large green, small green, split red). Fasting and postprandial blood samples were analyzed for glucose and insulin, and used to derive incremental area under the curve (iAUC), RGR, and maximum concentration (Cmax). Treatment effects were assessed with the use of repeated-measures ANOVA within the rice and potato treatments. Results: In comparison to rice alone, blood glucose iAUC and Cmax (P < 0.001) were lowered after consumption of rice with large green (P = 0.057), small green (P = 0.002), and split red (P = 0.006) lentils. Blood glucose iAUC and Cmax were also significantly lowered (P < 0.0001) after consumption of potato combined with each lentil, compared to potato alone. Plasma insulin iAUC and Cmax were significantly (P < 0.001) decreased when lentils were combined with potato, but not with rice. The RGRs of rice and potato were lowered by ∼20% and 35%, respectively, when half of their AC was replaced with lentils. Conclusions: Replacing half of the AC from high-GI foods with lentils significantly attenuates PBGR in healthy adults; this can contribute to defining a health claim for pulses and blood glucose lowering. This trial was registered at clinicaltrials.gov as NCT02426606.